use pipenv in image building dockerfile #36

jimexist · 2018-02-15T15:14:02Z

ready for review now

googlebot · 2018-02-15T15:14:08Z

So there's good news and bad news.

👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there.

😕 The bad news is that it appears that one or more commits were authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request.

Note to project maintainer: This is a terminal state, meaning the cla/google commit status will not change from this State. It's up to you to confirm consent of the commit author(s) and merge this pull request when appropriate.

jlewi · 2018-02-15T18:40:21Z

Is this ready for review? I ask because of the TODO? If not please put wip in the title.

jlewi · 2018-02-15T18:41:42Z

images/Dockerfile

@@ -92,13 +92,16 @@ RUN cd /tmp && \
    mv ks_0.8.0_linux_amd64/ks /usr/local/bin && \
    chmod a+x /usr/local/bin/ks

-COPY ./requirements.txt /tmp/requirements.txt
+COPY ./Pipfile ./Pipfile.lock /tmp/


Does this get used anywhere?

it is used in https://github.com/kubeflow/testing/pull/36/files#diff-475e714b0c3678141c59dab9961e8841R103

jimexist · 2018-02-25T06:27:19Z

@jlewi can you try to toggle and enable https://travis-ci.org/kubeflow/testing?

jlewi · 2018-02-27T14:35:59Z

@jimexist Sorry missed your first ping. Travis should be enabled now.

jlewi · 2018-02-27T14:37:06Z

@jimexist Can you create instructions for updating/generating the Pipfile.

jimexist · 2018-02-28T03:33:33Z

@jlewi will do. by the way any way to update the cla flag?

jimexist · 2018-02-28T04:10:31Z

nevermind - i squashed locally

jimexist · 2018-02-28T04:13:20Z

@jlewi seems CLA bot didn't re-check. Can you help manually merge?

this is cherry-picked from #36 so that it can unblock others.

jlewi · 2018-02-28T14:24:23Z

@jimexist Can you update the Makefile so that you can build/push an image without tagging it latest? And then add a separate build command to tag it latest?

Then can you build/push an image to kubeflow-ci.
Then i can quickly check that the dependencies are working.

jlewi · 2018-02-28T14:34:09Z

/assign jlewi

jlewi · 2018-03-01T18:28:26Z

@jimexist Were you able to test this image? If you wanted to you could create an E2E test (an Argo workflow) that would build the image and then do a smoke test on that image just by running a simple python script that would try to import various dependencies.

jlewi · 2018-03-20T02:45:08Z

Any update?

… process. (kubeflow#47) We want to use a different image for security reasons because test code running in the test project could potentially modify the test worker image. To deal with kubeflow#46 remove hashes from requirements.txt

…w#57) fixes kubeflow#54

* Replace sha256sum because it doesn't work on OSX

jimexist · 2018-03-20T03:28:01Z

/retest

jlewi · 2018-03-20T13:25:16Z

Thanks for updating it. Can you update the Makefile so I can try building an image and verify it works?

jlewi · 2018-03-27T12:57:24Z

@jimexist the Makefile was most likely updated in another file.
Can you resolve the conflicts?

jimexist · 2018-03-28T01:09:49Z

@jlewi updated, could you take a look?

jlewi · 2018-03-28T03:53:41Z

py/kubeflow/testing/util.py

@@ -47,7 +46,7 @@ def run(command, cwd=None, env=None, polling_interval=datetime.timedelta(seconds
      lines.append("{0}={1}".format(k, env[k]))
    logging.info("Running: Environment:\n%s", "\n".join(lines))

-  log_file = None
+  # log_file = None


Can you just delete this line?

jlewi · 2018-03-28T03:55:23Z

Can you do
make push
to build a new image (but not tag it latest) and then

Can you check that the python packages can be imported
Post a link to the image here?

IIRC in a previous attempt there was a problem with pipenv not actually installing the packages so it would be good to verify its working now.

jimexist · 2018-03-28T08:59:38Z

docker history gcr.io/kubeflow-ci/test-worker:v20180328-01bb4bc-e3b0c4
IMAGE               CREATED              CREATED BY                                      SIZE                COMMENT
3a7c875069b3        48 seconds ago       /bin/sh -c #(nop)  ENTRYPOINT ["/usr/local/b…   0B
3126fcc433b1        48 seconds ago       /bin/sh -c #(nop)  ENV PYTHONPATH=/src/kubef…   0B
a438cc458dd3        48 seconds ago       /bin/sh -c #(nop)  ENV USER=root                0B
ec4115b77de0        48 seconds ago       /bin/sh -c curl  https://get.docker.com/ | sh   185MB
52def2fd6e26        About a minute ago   /bin/sh -c chmod a+x /usr/local/bin/run_work…   520B
0de0b276dbea        About a minute ago   /bin/sh -c #(nop) COPY file:7335db716fdba146…   520B
3dcd1edd68e8        About a minute ago   /bin/sh -c chmod a+x /usr/local/bin/checkout…   2.76kB
6df67c4e7ee4        About a minute ago   /bin/sh -c #(nop) COPY file:d51a70df6dc36afb…   2.76kB
9ba7ac0ed83f        About a minute ago   /bin/sh -c cd /tmp/ &&     pip install -U pi…   244MB
bdc9c5752ed9        2 minutes ago        /bin/sh -c #(nop) COPY multi:848bf0c8dcf5bcf…   20.4kB
8332bcd7aafe        2 minutes ago        /bin/sh -c cd /tmp &&     wget https://githu…   10.3MB
d0c439ae8bbb        3 minutes ago        /bin/sh -c cd /tmp &&     wget -O ks.tar.gz …   49.3MB
9c87f1d5fd47        3 minutes ago        /bin/sh -c cd /tmp &&     wget -O glide-v0.1…   14.4MB
210187e5dc3c        4 minutes ago        /bin/sh -c curl -sS http://dl.yarnpkg.com/de…   6.18MB
60481f2a0a57        4 minutes ago        /bin/sh -c curl -sL https://deb.nodesource.c…   101MB
c5b92598233a        4 minutes ago        /bin/sh -c helm init --client-only              1.41MB
e9f542d9786b        4 minutes ago        /bin/sh -c wget -O /tmp/get_helm.sh     http…   64MB
977f59d7bbbb        4 minutes ago        /bin/sh -c wget -q https://dl.google.com/dl/…   366MB
169c06dcd53b        5 minutes ago        /bin/sh -c #(nop)  ENV PATH=/usr/local/go/bi…   0B
2d748b2b67fc        5 minutes ago        /bin/sh -c cd /tmp &&     wget -O /tmp/go.ta…   402MB
2320f38cc33f        5 minutes ago        /bin/sh -c set -ex     && apt-get update -yq…   378MB
b10bf3addc52        6 minutes ago        /bin/sh -c #(nop)  ENV LANG=C.UTF-8             0B
ad8d529b2fed        6 minutes ago        /bin/sh -c #(nop)  ENV LC_ALL=C.UTF-8           0B
361dbd1c5920        6 minutes ago        /bin/sh -c #(nop)  ENV TERM=linux               0B
2a9dbdd5af37        6 minutes ago        /bin/sh -c #(nop)  ENV DEBIAN_FRONTEND=nonin…   0B
1c15a190310f        6 minutes ago        /bin/sh -c #(nop)  MAINTAINER Jeremy Lewi       0B
f975c5035748        3 weeks ago          /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>           3 weeks ago          /bin/sh -c mkdir -p /run/systemd && echo 'do…   7B
<missing>           3 weeks ago          /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$…   2.76kB
<missing>           3 weeks ago          /bin/sh -c rm -rf /var/lib/apt/lists/*          0B
<missing>           3 weeks ago          /bin/sh -c set -xe   && echo '#!/bin/sh' > /…   745B
<missing>           3 weeks ago          /bin/sh -c #(nop) ADD file:c753df38640ab6e24…   112MB

jimexist · 2018-03-28T09:00:45Z

docker run --rm -it --entrypoint python gcr.io/kubeflow-ci/test-worker:v20180328-01bb4bc-e3b0c4 -V
Python 2.7.12

jimexist · 2018-03-28T09:00:52Z

docker run --rm -it --entrypoint python3 gcr.io/kubeflow-ci/test-worker:v20180328-01bb4bc-e3b0c4 -V
Python 3.5.2

jimexist · 2018-03-28T09:01:41Z

docker run --rm -it --entrypoint python3 gcr.io/kubeflow-ci/test-worker:v20180328-01bb4bc-e3b0c4
Python 3.5.2 (default, Nov 23 2017, 16:37:01)
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import kubernetes
>>> kubernetes.__version__
'4.0.0'

jimexist · 2018-03-28T09:08:30Z

@jlewi please take a look thanks

jlewi · 2018-03-28T13:20:52Z

Fantastic thank you so much.
/lgtm
/approve

k8s-ci-robot · 2018-03-28T13:20:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jlewi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jlewi]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jlewi · 2018-03-28T13:21:28Z

Merging manually because of the CLA check.

jlewi reviewed Feb 15, 2018

View reviewed changes

k8s-ci-robot added the size/L label Feb 17, 2018

jimexist changed the title ~~Use pipenv?~~ use pipenv Feb 25, 2018

jlewi mentioned this pull request Feb 27, 2018

Building the test worker image fails. #46

Closed

jimexist mentioned this pull request Feb 28, 2018

Create a Docker image to use as our workers for releases. #47

Merged

This was referenced Feb 28, 2018

use 2to3 to convert from Python2 to 3 and use yapf to reformat #35

Closed

Enable travis and do configurations #48

Merged

jimexist force-pushed the use-pipenv branch from 858ad5a to 738e22a Compare February 28, 2018 04:10

jlewi pushed a commit that referenced this pull request Feb 28, 2018

Enable travis and do configurations (#48)

2a12725

this is cherry-picked from #36 so that it can unblock others.

use pipenv to install packages and remove requirements.txt

87b0e90

jimexist force-pushed the use-pipenv branch from 55cc4cc to 87b0e90 Compare February 28, 2018 04:30

jimexist changed the title ~~use pipenv~~ use pipenv in image building dockerfile Feb 28, 2018

k8s-ci-robot assigned jlewi Feb 28, 2018

Merge branch 'master' into use-pipenv

643eced

jlewi and others added 4 commits March 20, 2018 11:09

Create an initial OWNERS file. (kubeflow#50)

e58af5d

Upgrade Argo so we can use DAGs. (kubeflow#51)

b6f0f91

change testing-argo.kubeflow.io to testing-argo.kubeflow.org (kubeflo…

6b4e974

…w#57) fixes kubeflow#54

Ankush Agarwal and others added 2 commits March 20, 2018 11:10

Add jsonnet to test-worker (kubeflow#68)

9c80289

* Replace sha256sum because it doesn't work on OSX

make waiting time configurable (kubeflow#69)

9f70d54

k8s-ci-robot added size/XXL and removed size/L labels Mar 20, 2018

Jiayu Liu added 2 commits March 20, 2018 11:14

fix up

5099377

Merge branch 'master' into use-pipenv

8738094

k8s-ci-robot added size/L and removed size/XXL labels Mar 20, 2018

jimexist added 2 commits March 27, 2018 23:37

Merge branch 'master' into use-pipenv

c898d9c

fix pylint issues

9a3ed36

jlewi reviewed Mar 28, 2018

View reviewed changes

remove extra line

01bb4bc

k8s-ci-robot added the lgtm label Mar 28, 2018

k8s-ci-robot added the approved label Mar 28, 2018

jlewi merged commit 4f8a605 into kubeflow:master Mar 28, 2018

jimexist deleted the use-pipenv branch March 28, 2018 13:50

pdmack mentioned this pull request Apr 23, 2018

minikube VM unavailable for e2e kubeflow-presubmit check #112

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use pipenv in image building dockerfile #36

use pipenv in image building dockerfile #36

jimexist commented Feb 15, 2018 •

edited

Loading

googlebot commented Feb 15, 2018

jlewi commented Feb 15, 2018

jlewi Feb 15, 2018

jimexist Feb 25, 2018

jimexist commented Feb 25, 2018

jlewi commented Feb 27, 2018

jlewi commented Feb 27, 2018

jimexist commented Feb 28, 2018

jimexist commented Feb 28, 2018

jimexist commented Feb 28, 2018

jlewi commented Feb 28, 2018

jlewi commented Feb 28, 2018

jlewi commented Mar 1, 2018

jlewi commented Mar 20, 2018

jimexist commented Mar 20, 2018

jlewi commented Mar 20, 2018

jlewi commented Mar 27, 2018

jimexist commented Mar 28, 2018

jlewi Mar 28, 2018

jlewi commented Mar 28, 2018

jimexist commented Mar 28, 2018

jimexist commented Mar 28, 2018

jimexist commented Mar 28, 2018

jimexist commented Mar 28, 2018 •

edited

Loading

jimexist commented Mar 28, 2018

jlewi commented Mar 28, 2018

k8s-ci-robot commented Mar 28, 2018

jlewi commented Mar 28, 2018

use pipenv in image building dockerfile #36

use pipenv in image building dockerfile #36

Conversation

jimexist commented Feb 15, 2018 • edited Loading

googlebot commented Feb 15, 2018

jlewi commented Feb 15, 2018

jlewi Feb 15, 2018

Choose a reason for hiding this comment

jimexist Feb 25, 2018

Choose a reason for hiding this comment

jimexist commented Feb 25, 2018

jlewi commented Feb 27, 2018

jlewi commented Feb 27, 2018

jimexist commented Feb 28, 2018

jimexist commented Feb 28, 2018

jimexist commented Feb 28, 2018

jlewi commented Feb 28, 2018

jlewi commented Feb 28, 2018

jlewi commented Mar 1, 2018

jlewi commented Mar 20, 2018

jimexist commented Mar 20, 2018

jlewi commented Mar 20, 2018

jlewi commented Mar 27, 2018

jimexist commented Mar 28, 2018

jlewi Mar 28, 2018

Choose a reason for hiding this comment

jlewi commented Mar 28, 2018

jimexist commented Mar 28, 2018

jimexist commented Mar 28, 2018

jimexist commented Mar 28, 2018

jimexist commented Mar 28, 2018 • edited Loading

jimexist commented Mar 28, 2018

jlewi commented Mar 28, 2018

k8s-ci-robot commented Mar 28, 2018

jlewi commented Mar 28, 2018

jimexist commented Feb 15, 2018 •

edited

Loading

jimexist commented Mar 28, 2018 •

edited

Loading